Skip to content

Schedules as automations - #2293

Open
Flix6x wants to merge 25 commits into
feat/2288-automations-for-forecastsfrom
feat/2288-schedule-automations
Open

Schedules as automations#2293
Flix6x wants to merge 25 commits into
feat/2288-automations-for-forecastsfrom
feat/2288-schedule-automations

Conversation

@Flix6x

@Flix6x Flix6x commented Jul 11, 2026

Copy link
Copy Markdown
Member

Description

Automations can now compute schedules on a recurring basis, alongside forecasts. The automation
still only decides when work is due and queues it; the existing scheduling machinery computes the
schedule and records it.

Parameters are a trigger message. A schedule automation stores what the
[POST] /assets/(id)/schedules/trigger endpoint accepts, without the asset id, validated against
AssetTriggerSchema when the automation is created. Use the canonical API field names, including
flex-model, flex-context and force-new-job-creation.

The schedule moves with the clock. Omit start and each run schedules from its own run time,
floored to the message's resolution when given and otherwise to the minute. A fixed start is
accepted, but then every run computes the same period, so the CLI warns when creating such an
automation.

Minimal triggers inherit stored flex config. The flex-model and flex-context can live on the
asset tree instead of in the trigger message, so duration: "PT12H" on its own is a complete
automation for an asset that already describes its own flexibility. The scheduler resolves the rest
the same way it does for a triggered schedule.

No data generator. A forecast automation points at a data source holding its forecaster
configuration; a schedule automation has nothing equivalent, so generator_id is now nullable and a
database constraint requires it only for forecasts.

Provenance. Scheduling jobs record how they came about in the same way forecasting jobs do:
{"origin": "automation", "automation_id": <id>}, so a scheduled job queued by an automation is
distinguishable from one triggered through the API or the CLI. Job statistics count scheduling jobs
for schedule automations, including the per-device jobs of a sequential schedule.

CLI. flexmeasures add automation --type schedules takes the trigger message through
--parameters. Forecaster options are refused for schedule automations — the check asks which
options were actually given on the command line, rather than comparing their values against their
defaults, so naming the default forecaster is refused too instead of passing silently. A
--parameters or --config file that is not a mapping is reported as such rather than failing
obscurely later.

UI. The Schedules tab on an asset's Automations page is now populated, with one listing per
automation type. Reports remain a disabled placeholder until #2297.

  • Added changelog item in documentation/changelog.rst

Look & Feel

The Schedules tab is now populated, with its own listing alongside the forecasts one. Here an hourly
schedule automation on a battery, recurring in Europe/Amsterdam. Reports remains a placeholder until
#2297:

The Automations page with the Schedules tab selected, listing an hourly schedule automation on a battery, with the Reports tab still disabled

How to test

See the manual test walkthrough in the PR comments.

pytest \
  flexmeasures/cli/tests/test_automations.py \
  flexmeasures/data/tests/test_automations_fresh_db.py \
  flexmeasures/data/tests/test_automation_scheduling_fresh_db.py \
  flexmeasures/api/v3_0/tests/test_automations_api.py \
  flexmeasures/api/v3_0/tests/test_automations_api_fresh_db.py \
  flexmeasures/ui/tests/test_asset_crud.py

Further improvements

  • Which sensors a schedule automation reads and writes is worked out by resolving the flex config
    and then looking for the fields that name where results go. That agrees with the scheduler's own
    resolution for the shapes that matter — a device's sensor, and its consumption and
    production outputs — and errs towards reporting more rather than fewer, which is the safe
    direction once the same resolution guards who may create an automation (CRUD for automations in the API and UI #2294). Checking outputs
    against what a scheduler actually returns at run time is tracked in Check the sensors a scheduler actually writes to, instead of predicting them when an automation is created #2421.
  • Catch-up policy is defined for forecasts. What it should mean for a schedule — whether a missed
    occurrence is worth scheduling at all once its window has passed — is still open.

Related items

Closes #2370. Part of the automations story #2334. Stacked on #2290, followed by #2294.


Sign-off

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on code under GPL or another incompatible license.

Flix6x and others added 3 commits July 11, 2026 18:29
The scheduling job creators accept an optional trigger dict (stored as job
meta data), like the forecasting pipeline already does. The API trigger
endpoint records origin API; the CLI and automations follow in the next
commit. The status page's 'Created Via' column picks this up automatically.

Part of #2288

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Automations now also support the 'schedules' type:

- `flexmeasures add automation --type schedules` validates the parameters as
  a schedule trigger message (per the AssetTriggerSchema, as accepted by the
  API trigger endpoint, without the asset id). The schedule 'start' may be
  omitted, in which case each run schedules from the run time (floored to the
  message's resolution, if given) — a fixed start draws a warning.
- The runner dispatches schedules automations to the same job creators as the
  API trigger endpoint (sequential or simultaneous), recording trigger meta
  data (origin automation) on the queued jobs; `flexmeasures add schedule
  --as-job` now records origin CLI.
- Job stats for schedules automations are counted from the scheduling job
  cache (asset-level wrap-up jobs and per-sensor device jobs).
- The UI automations page's Schedules tab is now enabled, with automations
  filtered by type per tab.

Part of #2288

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
@BelhsanHmida BelhsanHmida linked an issue Jul 30, 2026 that may be closed by this pull request
BelhsanHmida and others added 16 commits August 5, 2026 12:01
Context:
- PR #2293 is stacked on the reviewed forecast automation branch.

Change:
- Merge the reviewed parent and preserve schedule automation behavior across the five semantic conflicts.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context:
- Schedule automations do not use a data generator, but the reviewed forecast automation schema required one.

Change:
- Make the foreign key nullable while retaining a database check that forecasts always have a generator.
- Add a forward migration without weakening data-source deletion semantics.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context:
- Review uncovered untested forecast-only options, invalid durations, and DST start calculation.

Change:
- Add CLI and trigger-preparation regressions while retaining forecast sensor validation.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context:
- Persistence, inherited flex configuration, descendant statistics, and job counts lacked realistic coverage.

Change:
- Move automation service tests under the fresh-database fixture and add end-to-end schedule regressions.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context:
- API provenance was not asserted across asset and sensor schedule jobs.

Change:
- Verify API trigger metadata on sequential descendants, wrap-up jobs, and sensor jobs.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context:
- The asset automation page tests still targeted the former single table.

Change:
- Assert type-specific tables, error rendering, filters, and hidden-tab column adjustment.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context:
- Minimal asset schedules lost stored defaults, invalid timing could execute, provenance was incomplete, and descendant jobs were miscounted.

Change:
- Validate and floor fixed durations safely, inherit stored flex configuration, preserve provenance, and count the jobs actually dispatched.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context:
- Schedule automation creation silently accepted forecaster settings that could never affect scheduling.

Change:
- Detect supplied forecast-only options and return a user-facing usage error.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context:
- DataTables initialized in the hidden schedule tab could render with stale column widths.

Change:
- Add tab accessibility state and adjust initialized table columns when a tab becomes visible.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context:
- The feature guide linked only to the API root and omitted fixed-start and duration constraints.

Change:
- Document canonical fields, runtime start behavior, timing validation, and generator-free schedule automations.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context:
- Manual testing exposed raw parser exceptions for malformed automation files.

Change:
- Require a user-facing usage error for invalid YAML in both config and parameter files.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context:
- PyYAML parser errors escaped automation creation without a useful CLI message.

Change:
- Translate malformed config and parameter files into a normal Click usage error.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context:
- Manual execution showed minimal automations failing for a one-device asset tree.

Change:
- Exercise both simultaneous and sequential dispatch using realistic flex configuration stored on the child asset.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context:
- A one-device asset tree collapsed to sensor scheduling without a sensor, and sequential dispatch could not resolve its stored output.

Change:
- Preserve asset-triggered flex models as a list and resolve sequential device sensors from stored consumption or production outputs.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context:
- Scheduling service docstrings did not distinguish single-job and sequential provenance behavior.

Change:
- Document where trigger metadata is stored and correct the simultaneous return description.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Brings in the timezone and catch-up work from #2396, the sensor links, the sortable listing and the review fixes from #2290.

Resolved by keeping both sides, so an automation carries a timezone and a scheduling cursor as well as its input and output sensors,
and the job statistics still scan the scheduling queue for schedule automations while the forecast path is unchanged.

Two resolutions went further than picking a side.
make_cli_options_optional is gone: add_cli_options_from_schema now takes force_optional, which relaxes the same requirement and also hides the forecast options,
which is what a schedule automation wants anyway, since those options only apply to forecasts.
The asset page test no longer expects a single automations table to be hidden, as the listing is now one table per automation type.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@BelhsanHmida
BelhsanHmida marked this pull request as ready for review August 10, 2026 23:33
BelhsanHmida and others added 5 commits August 11, 2026 01:07
Merging the timezone and catch-up work alongside the schedule automations left two alembic heads,
one adding an automation's timezone and scheduling cursor and one allowing a schedule automation without a data generator,
so flexmeasures db upgrade refused to run and the Docker image build failed.
The two touch different columns, so the merge point has nothing of its own to do.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>

# Conflicts:
#	flexmeasures/data/services/automations.py
Prepare the scheduling configuration through the same scheduler collection path used before queueing, then derive declared input and output sensors so generator-free schedule automations expose their stored flex dependencies and participate in sensor relationships.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Exercise a minimal schedule that inherits its flex model and context from the asset tree, confirming that price inputs, schedule outputs, and the sensor-to-automation relationship are all reported from stored configuration.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Create a generator-free schedule with stored price and output sensors and assert that the details endpoint returns both dependency sets instead of reporting empty arrays.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
…differ from their default

The guard against combining forecaster options with --type schedules compared the forecaster against its default,

so naming the default forecaster explicitly passed silently and the automation was created,

leaving the impression that the option had applied to a schedule automation.

Ask which options were actually given on the command line instead, the way the --source conflict check already does,

and name the offending options in the error rather than listing every option it could have been.

Signed-off-by: Mohamed Belhsan Hmida mohamedbelhsanhmida@gmail.com
@BelhsanHmida

Copy link
Copy Markdown
Contributor

Manual test walkthrough

Everything below was run against a development database on this branch. It assumes the forecast
roundtrip from #2290 already makes sense; this covers only what schedules add.

Pick an asset that describes its own flexibility, so a minimal trigger has something to work with —
a battery with a stored flex-model is ideal. The examples use asset 25, a charge point whose
flex-model is {soc-max: 56 kWh, soc-min: 13.02 kWh, power-capacity: 5.75 kW}.

flexmeasures jobs run-worker --name schedules-demo --queue scheduling

1. A minimal trigger is enough

The parameters of a schedule automation are a trigger message: what
[POST] /assets/(id)/schedules/trigger accepts, without the asset id. When the asset already
carries its flex config, the message can be one line.

echo 'duration: "PT12H"' > trigger-message.yml

flexmeasures add automation \
  --asset 25 \
  --name "Hourly schedules" \
  --cron "0 * * * *" \
  --timezone Europe/Amsterdam \
  --type schedules \
  --inactive \
  --parameters trigger-message.yml
# → Successfully created inactive automation 'Hourly schedules' (ID: N) to compute schedules
#   for asset 25, recurring per cron string '0 * * * *' in timezone 'Europe/Amsterdam'.

Note it needs no forecaster and no data source: generator_id is null for schedule automations, and
a database constraint requires a generator only for forecasts.

2. The schedule starts when the run starts

Omitting start is the point of a recurring schedule. Each run fills it in from its own run time,
floored to the message's resolution if given and otherwise to the minute, so the stored
{"duration": "PT12H"} becomes, at run time:

{'duration': 'PT12H', 'id': 25, 'start': '2026-08-12T22:53:00+00:00'}

A fixed start is accepted, but then every run computes the same window, so the CLI says so:

echo 'start: "2025-11-16T00:00:00+01:00"' >  trigger-fixed.yml
echo 'duration: "PT12H"'                  >> trigger-fixed.yml

flexmeasures add automation --asset 25 --name "Fixed window" --type schedules \
  --inactive --parameters trigger-fixed.yml
# → Warning: the schedule 'start' is fixed, so each run will compute the same period.
#   Omit 'start' to schedule from the run time instead.
# → Successfully created inactive automation 'Fixed window' (ID: M) ...

3. Forecaster options do not apply

A schedule automation is not computed by a forecaster, so anything configuring one is refused and
names the option at fault:

flexmeasures add automation --asset 25 --name "Wrong options" --type schedules \
  --parameters trigger-message.yml --forecaster SomeOtherForecaster
# → Error: --forecaster cannot be combined with --type schedules: a schedule automation is not
#   computed by a forecaster.

flexmeasures add automation --asset 25 --name "Wrong options" --type schedules \
  --parameters trigger-message.yml --train-period P7D
# → Error: --train-period cannot be combined with --type schedules: ...

Naming the default forecaster is refused as well, since it was still given:

flexmeasures add automation --asset 25 --name "Wrong options" --type schedules \
  --parameters trigger-message.yml --forecaster TrainPredictPipeline
# → Error: --forecaster cannot be combined with --type schedules: ...

The check asks which options were passed rather than comparing their values against their defaults,
so an option cannot slip through by happening to match the default.

A parameters file that is not a mapping is reported plainly rather than failing later:

echo 'just a string, not a mapping' > bad.yml
flexmeasures add automation --asset 25 --name "Bad yaml" --type schedules --parameters bad.yml
# → Error: The --parameters file must contain a YAML or JSON object at the top level.

4. Run it, and see a scheduling job

flexmeasures edit automation --id N --cron "* * * * *" --activate
flexmeasures jobs run-automations
# → Automation N ('Hourly schedules') queued 1 scheduling job(s) for asset 25.

The job lands on the scheduling queue, not the forecasting one, and carries the same provenance
a forecast automation gives its jobs:

queued on queue : scheduling
job trigger meta: {'origin': 'automation', 'automation_id': N}
asset_or_sensor : {'id': 25, 'class': 'Asset'}

For a sequential schedule the automation queues one job per device plus the wrap-up job, and the job
statistics on the automations page count all of them.

5. The Schedules tab

/assets/25/automations

The Schedules tab is now populated, alongside Forecasts. Each type has its own listing, and the
tables resize correctly when switching between tabs. Reports stays disabled until #2297.

6. Clean up

flexmeasures edit automation --id N --deactivate
flexmeasures delete automation --id N --force

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Schedules as automations

2 participants